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ABSTRACT 
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to be used with guidance from the National Skill Standards Board, it is 
designed for the voluntary partnerships that have completed the standards 
development phase and are ready to begin the phases of assessment and 
certification (A&C) . An introduction outlines the work ahead; examines the 
key activities involved in each of the three major components of the work 
(learn and prepare; develop the system; and launch the system) ; discusses 
working with vendors; and lists keys to success. The rest of the manual 
describes the step-by-step approach to developing the first component. These 
are the four steps: (1) organize- -building a strong organizational structure 

to tackle issues involved with A&C; (2) investigate- -learning about A&C, the 
customer, the market, civil rights and access concerns, and other key issues; 
(3) envision- -creating a vision of how the A&C system should work; and (4) 
plan and retain- -retaining expert assistance to plan, develop, administer, 
and maintain the system. (Contains 88 references.) (YLB) 
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About This Manual... 



Getting Started on Assessment provides practical advice for building a system of 
assessment and certification based on skill standards. This manual was devel- 
oped for the Voluntary Partnerships who have completed the Standards 
Development Phase and are ready to begin the Phases of Assessment and 
Certification. It should be used with guidance from the National Skill 
Standards Board (NSSB). 

Who We Are... 

The National Skill Standards Board (NSSB) is a coalition of leaders from busi- 
ness, labor, employee, education, community, and civil rights organizations cre- 
ated in 1994 to build a voluntary national system of skill standards, assessment, 
and certification. The goal of this effort is to ensure American workers have the 
skills they need to succeed in today’s workplace, enhancing U.S. global compet- 
itiveness and raising the living standard of all Americans. NSSB-endorsed skill 
standards, which form the foundation of this new system, are being identified 
by Voluntary Partnerships, industry coalitions working in full partnership with 
labor, civil rights, and community-based organizations. 

To Find Out More, Contact Us... 

On the Web at www.nssb.org 
By email at information@nssb.org 

By phone at (202) 254-8628 or (877) THE-NSSB 

By fax at (202) 254-8646 

By mail at National Skill Standards Board 
1441 L Street, NW, Suite 9000 
Washington, DC 20005-3512 



Edition 1, March 2001 
© 2000 National Skill Standards Board 

National Skill Standards Board 
“ Setting the Standard for Workforce Excellence ” 
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M, 



Introduction 




Now that your skill standards have been developed and approved, your 
Voluntary Partnership will begin developing a system of assessment and certifi- 
cation based on skill standards. This is an exciting time — your first chance to 
put skill standards into practice on a large scale. This manual is meant to help 
you get started. 

Planning for assessment and certification 
requires much of the same solid research, 
innovative thinking, and practicality that 
goes into developing a top-quality prod- 
uct and bringing it to market. 

Who are your customers? What are their 
needs? How can you create a product or 
service that will succeed in today’s mar- 
ketplace? As you begin to build an 
assessment and certification system, 
answering these kinds of questions will 
be critical. 



Before Reading 
this Manual... 

Please read Credentials 
for Success, which out- 
lines the NSSB's vision 
of assessment and certi- 
fication. It includes 
valuable information 
about assessment and 
certification not cov- 
ered in this manual. 



The Work Ahead 



The best way to begin building your assessment and certification system is with 
a realistic sense of the work ahead. Each Voluntary Partnership will be responsi- 
ble for a range of activities, including: 



A Developing the assessments; 

A Setting up assessment delivery channels; 

A Making key policy decisions regarding the certification program (e.g., 
how much to charge, how to ensure accessibility) ; 

A Complying with the NSSB’s criteria for assessment and certification as 
well as legal, technical and documentation requirements; 
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A Establishing a national database for storing and retrieving information; 

A Making plans for administration and ongoing maintenance of the system; 

and 

A Marketing the system. 

Even a quick look through this partial list hints at the magnitude of the work 
ahead. Your Voluntary Partnership will not be tackling these challenges alone. 
The NSSB and its Technical Assistance Team will provide advice and assistance 
to help your Voluntary Partnership. The NSSB also recommends that 
Voluntary Partnerships enlist the help of an outside vendor or vendors to 
design and implement the system. To enhance your internal capabilities, you 
may also wish to hire an assessment expert to serve on staff. 



Three Major Components of the Work 

Because so much work is involved in building a certification system, which 
includes standards and assessment, the NSSB will be providing technical assis- 
tance in a phased approach: 






In the following section, we will examine the key activities involved in each 
major component of development. 
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A Look at Each Component 




The First Component involves laying the foundation for the development of your assess- 
ment and certification system. This includes building your own knowledge of assessment 
and certification, beginning to learn about your customers, and obtaining the support 
of an outside vendor or vendors. Here’s a look at the major activities involved: 

H Learn the basics about assessment and certification. Now is the time to 
build your own knowledge of assessment and certification and the related 
NSSB criteria. This will help you manage the work ahead. The NSSB will 
assist you through a series of workshops, publications, etc. 

ES Build internal technical capabilities. Once you understand the basics, 
begin building your assessment team. The NSSB recommends that you start 
by hiring someone who can help you manage the process (e.g., prepare the 
request for proposals (RFP) , select and oversee vendors) . 

11 Conduct market research. Take time to learn more about the market for 
assessment and certification in your industry sector. A strong understand- 
ing of your customers and the market will help you make critical decisions 
about how to deliver your assessments, how much to charge, and so on. 
You may choose to do this market research on your own or with the help 
of outside experts. Be sure to build on earlier market research conducted 
when your Voluntary Partnership was first established. 

m Start outreach efforts. At this early stage, it is important to start forming 
relationships with specialty skill standards developers, colleges, companies, 
unions, training organizations, state governments, civil rights and commu- 
nity-based organizations and others that can help promote and support 
your system. The NSSB is working to build relationships with these groups 
on a national, cross-sector level and can offer you support and advice. 

H Develop a marketing plan. It is going to take time, effort, and a fairly 
sophisticated marketing effort to persuade employers and workers to use this 
new system. Start thinking about this issue as early as possible. The NSSB 
may be able to provide support in this area — a coordinated marketing 
strategy among the Voluntary Partnerships will double or triple the impact. 
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El Further develop your business 
plan. Consider how your 
Voluntary Partnership will finance 
the system, deliver products and 
services, and the like. Your 
Voluntary Partnership has already 
developed an overall business plan, 
which may have even addressed 
issues specific to offering assess- 
ment and certification services. 

Now is the time to refine that plan. 

Talk to NSSB staff and other 
Voluntary Partnerships about ways 
to leverage resources. Joining forces 
may help you create high-quality 
products and services and get them 
to market faster. 

■ Identify individuals with expert- 
ise in employment law, access, 
diversity, and civil rights issues. 

It will be important to find indi- 
viduals with knowledge of alterna- 
tive methods of assessment 
designed to minimize adverse 
impact while maintaining the 
validity of the assessments. 

Incorporating issues of access and civil rights is vitally important and 
should be factored in as early as possible. The NSSB will provide technical 
support in this area. 

■ Prepare an RFP and Criteria for Evaluating Proposals. 

With your internal assessment team created, it’s time to issue 
an RFP to enlist the help of an outside vendor or vendors. 
When you develop your RFP, you should also develop crite- 
ria for evaluating proposals. You must submit your RFP to 
the Endorsement Review Panel (ERP) before issuing it. 




^SQgvaiuating 
Proposals 



Endorsement Review 

Next to certain activi- 
ties within each phase, 
the following icon will 
appear. 

This icons 




signals 
that 

the output of this activi- 
ty must be submitted to 
the NSSB Endorsement 
Review Panel (ERP): 

The ERP will review 
each of these deliver- 
ables to ensure that it 
meets key NSSB criteria. 
For more information 
about the NSSB criteria 
and endorsement 
process, please see: 
NSSB Endorsement 
Process: A User's Guide 
for Voluntary 
Partnerships. 
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The Second Component involves working with your vendor to build the system. 

This includes developing assessment tools, setting up delivery channels, conducting 
pilots, etc. At each stage, the NSSB will provide technical assistance to help you and 
your vendor develop the best possible plan. Let’s take a look at some of the important 
activities involved in this work: 

E3 Develop a Preliminary Assessment System Plan. This is 
the first step in developing assessments and involves the 
important and challenging task of determining which aspects 
of the skill standards will be assessed and the best methods 
for assessing them. The plan should provide information on 
the estimated importance assigned to each aspect of the standards, the 
proportion of the assessments that will be dedicated to each component of 
the skill standards, and an estimate of scoring and examination time. It 
should also provide a vision of how the various aspects of the plan will 
lead to certification, how the system will be delivered, what system com- 
ponents are needed, etc. Your Preliminary Assessment System Plan must be 
submitted to the ERP. 

H Create a Detailed System Development Plan. Your next 
step is to develop a detailed plan for developing your assess- 
ment system. The first aspect of the plan needs to deal with 
the assessment tool development. It should literally be a blue- 
print for creating the assessment tools. It should describe 
exactly how each aspect of the skill standards will be meas- 
ured, the types of questions candidates will be asked, the types of tasks they 
will be requested to perform, and so on. It should also include a linkage 
study, showing how what’s being measured and the way in which it is 
being measured accurately reflects the content of the skill standards. The 
second aspect of the plan needs to deal with the development of the system 
components. It should describe precisely how your Voluntary Partnership 
intends to deliver the assessment tools (e.g., delivery channels, providing 
candidate information and feedback) ; create the national database for stor- 
ing and retrieving information; develop a reporting mechanism for candi- 
dates and participating employer organizations; ensure secure, “hacker- 
proof” delivery, storage, and reporting systems; create a method for ascer- 
taining how different groups are performing (e.g., mean scores and stan- 
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dard deviations across different concentrations, performance of various 
protected groups, basic psychometric indices) 1 ; develop training programs 
for people delivering and servicing the system; provide pre-assessment 
information and services, etc. In all these efforts, you should incorporate 
the NSSB goal of building a voluntary national system of assessment and 
certification that increases opportunities for minorities and women. Your 
Detailed System Development Plan must be submitted to the ERP. 

@ Develop the assessment tools and conduct a preliminary 
evaluation of the assessments. In this step, your vendor will 
develop the assessment tools, including scoring keys and rating 
scales 2 for each tool. This stage also includes the development 
of algorithms for combining assessments from the different 
tools into an overall score. Another important part of this step 
is a preliminary evaluation of the assessment tools. You and your vendor will 
conduct pilots to gather preliminary data about reactions to the tools and 
psychometric characteristics 3 of the tools. Results of this preliminary evalua- 
tion must be submitted to the ERP. 




L Results of 
reliminary 
J valuation of 
Assessment 
Tools 



El Develop system components. This step includes establishing delivery 
channels (e.g., setting up sites to assess candidates or working with exist- 
ing ones); creating a national database for storing assessment results; 
developing a mechanism for reporting results to candidates and issuing 
certificates; addressing quality control and security issues regarding assess- 
ment administration, scoring keys and algorithms, and database access 
and integrity; developing training modules for assessment tool administra- 
tors, system administrators, assessors, and other staff; developing an ongo- 
ing maintenance plan for continuously improving the system and updat- 
ing it as skill standards change. 



r$a 




— — ^ 

£ 


. Plan for 
ull-Scale 
ystem 
valuation 



1 Develop a plan for full-scale evaluation of the entire 
system. The plan should describe in detail how you intend 
to evaluate all parts of the assessment system. This is in con- 
trast to the preliminary evaluation, which evaluated the 
assessment tools only. The plan for the full-scale evaluation 



1 This important aspect of developing your assessment system should be considered as part of the initial design of the sys- 
tem. If it is not, it could be costly and extremely difficult to ascertain this information. 

2 Scoring keys provide information about the right and wrong responses to more straightforward forms of assessments 
(e.g., a multiple choice test). Rating scales often include checklists for raters to use in evaluating a candidate’s performance 
and are often used for assessments involving work simulations or an evaluation of a work sample. 

3 Psychometric characteristics refers, among other things, to the reliability of the assessments (the degree to which the assess- 
ments accurately capture the content of the skill standards, measure that content, etc.). 
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should include information about how you plan to evaluate candidate 
reactions; participating employer organization reactions; scoring strategies; 
impact of assessments on protected groups; psychometric characteristics of 
the assessment tools; readability analyses; linkage of assessment tools with 
work standards; effectiveness of security measures and training programs 
for assessment administrators; accessibility of the system to users, effec- 
tiveness of national database, etc. Please Note: This evaluation needs to 
be conducted before the system is launched. 



f Results of 
ull-Scale 

-Jystem 



Evaluation 



B Conduct a full-scale evaluation of the assessment sys- 
tem and make changes as required. Now it’s time to con- 
duct a full-scale evaluation of the assessment system, includ- 
ing the assessment tools and system components. This evalu- 
ation should measure the effectiveness of all aspects of the 
system (e.g., psychometric characteristics of assessment tools, 
comparison of mean scores for groups of interest for all concentrations, 
scoring, delivery channels, storage and communication of results, security 
of the delivery system and databases). Based on findings, your assessment 
system will likely require modification before it is launched. 

■ Prepare a Technical Report documenting the content 
validity and effectiveness of the assessment system. This 
report should describe in detail the activities undertaken to 
develop and evaluate the assessment system. This report 



m 






. Technical 
eport 



should be sufficiently detailed to allow an expert to evaluate the quality of 
the system. It should conform to the Uniform Guidelines, the Principles 
and the Standards. The technical report should include the final assess- 
ment tools and components. 

Develop a rollout plan and fine-tune the marketing and business 
plan. How, when, and where will you launch the system? Will you start 
with six assessment centers in the first year and grow to a dozen in the 
next? Where is the greatest need for assessment in your industry sector? 
How will that affect your rollout plan? To whom will you market first? 
How will you ensure the rollout plan helps you achieve the important 
goal of increasing opportunities for minorities and women? How will you 
measure success? All these issues will need to be addressed. The NSSB and 
other Voluntary Partnerships will be a helpful source of information in 
developing your strategy. 
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The Third Component involves the initiation of services. At this time, your 
Voluntary Partnership should be launching its full-scale marketing plan; activating 
its ongoing maintenance plan, including the updating of standards and assessments 
and the ongoing evaluation of access and compliance issues; and working to link the 
system to specialty assessment and certification systems. Further information about 
ERP Deliverables will be available closer to the launch. 



The Focus of This Manual 

Getting Started on Assessment 
offers advice to help your 
Voluntary Partnership begin 
developing the First Component. Future publications will help you develop the 
Second Component and the Third Component. 

The NSSB has developed a step-by-step approach to help you get started on 
developing the First Component: 

4^ Step 1: Organize. Building a strong organizational structure to tackle 
the issues involved with assessment and certification. 

Step 2: Investigate. Learning about assessment and certification, your 
customer, the market, civil rights and access concerns, and other key 
issues. 

4^ Step 3: Envision. Creating a vision of how you want your assessment 
and certification system to work. 

4^ Step 4: Plan and Retain. Retaining expert assistance to plan, develop, 
administer, and maintain the system. 




This step-by-step approach is described in the rest of this guidebook. Your 
Voluntary Partnership may wish to use this approach as is or modify it to meet 
your needs. 



ERIC 
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Working with Vendors 

Because of the highly technical nature of 
assessment, the NSSB recommends that 
you work with an outside vendor or 
group of vendors to develop and imple- 
ment your assessment and certification 
system. Although it’s tempting to hire a 
vendor right away, we encourage your 
Voluntary Partnership to take the time 
to learn about assessment and certifica- 
tion, your customer, and what you want 
to achieve. A clear vision and a complete 
understanding of assessment will help 
you select the best contractors and help 
you manage them. 

In light of this, we recommend that you 
complete the first four steps of the First 
Component before obtaining the assis- 
tance of an outside vendor, the subject 
of Step Four (Plan and Retain) . 

The NSSB will be available to provide 
advice and assistance as you work 
through the first three steps of this 
process. When your Voluntary 
Partnership is ready, the NSSB would like to begin work on the steps described 
in this manual with an initial training and planning session with your 
Voluntary Partnership’s managers (directors) , assessment staff, and assessment 
committee. 



Benefits for 
Everyone 

This guidebook is 
geared for all Voluntary 
Partnerships, including 
newly formed partner- 
ships and those already 
established. Your 
Voluntary Partnership 
may have started on 
some of the steps? 
described in the early 
sections of this guide- 

| book. We encourage 

m*. - . ?• i m - ' ■ ■■■'« 

you to review these 

steps, looking for new 
ideas or lines of ques- 
f tioning that may help 
you refine your plans. 
Once you have finished 
your review, you may 
wish to start with some 
of the later steps in the 
guidebook. 
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Keys to Success 

As you begin this work, remember to: 

Think big. An assessment and certification system is much more than just the 
assessment methods used. It must address tough issues, from how to pay for the 
system, to how to link it to training and development, to how to ensure it com- 
plies with civil rights laws. 

Be creative. This is a new and challenging task that will require innovative 
thinking. Although there are many lessons to be learned from existing assess- 
ment efforts, you will be addressing many new issues based on the unique char- 
acteristics of national skill standards. 

Be realistic. Building a national system of assessment and certification means 
being realistic about what it can and cannot do: 

H It won’t sell itself. We cannot take the “if you build it, they will come” 
approach. Workers, employers, students, education and training 
providers, and others need to be sold on the value of assessment and certi- 
fication. This not only requires a strong marketing plan but also a com- 
mitment to developing a system that meets the needs of all those who will 
use the system, from individuals to employers to educators. 

B It won’t create itself. Building an assessment and certification system is a 
major undertaking that requires significant resources and work on the 
part of each Voluntary Partnership and the NSSB. Getting started as early 
as possible is critical to the development, planning, and implementation 
of the system. 

IP It won’t work if it’s not accessible. Assessment and certification must be 
affordable; geographically accessible; and practical in terms of the time, 
cost, and effort required of the individual. 

HI It won’t work if employers, employees, students, and educators don’t recognize, 
value, and use it. Assessment and certification will have huge implications 
for the careers of individuals, the success of companies, and the reputations 
of education and training providers. Spend time at the outset to build a 
system of the highest quality and of the most use to your stakeholders, and 
you can build the trust and respect that will be so critical for success. 
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Organize 

Building a strong organizational 
structure to tackle the issues 
involved with assessment and 
certification 



Developing assessment tools is really the last step in a comprehensive process 
that begins with the formation of your Voluntary Partnership and the develop- 
ment of skill standards. Spend time at the start to build a strong organizational 
structure to address assessment and certification issues. To do this, the NSSB 
recommends that you: 




Establish a Separate Assessment and 
Certification Committee 



Preparing for assessment and certification begins when you set up the opera- 
tional and committee structure that will guide your Voluntary Partnership. 
More than likely, you will establish separate committees to oversee different 
issues, such as standards development, marketing and communication, and 
others. Given the scope and complexity of assessment, we recommend that 
you consider establishing a separate assessment and certification committee . 4 



4 Your Voluntary Partnership may already have a committee structure set up. If you have a separate assessment and certifi- 
cation committee, begin using the ideas in this chapter to ensure the committee is on track. If you don’t have a separate 
committee, now is the time to establish one and address these critical issues. 
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The work of the assessment and certification committee should not be lim- 
ited to overseeing the development of the assessment tools. The committee 
should also: 

Make recommendations to your Voluntary Partnership’s Decision 
Council on ways to ensure equal access to the system, portability of cer- 
tificates across the industry sector, and other pertinent issues; 

Determine how best to serve customers; 

Help select and oversee the assessment and certification vendor; 

Build strategic relationships (e.g., evaluate the benefits of potential part- 
nerships and build relationships with education and training providers, 
specialty standards developers, and others who can help promote your 
system) ; 

Address practical issues such as the marketing, cost, delivery, and mainte- 
nance of your system; and 

Contribute to your Voluntary Partnership’s overall business plan. 

It’s important to set up a way for members of the Standards Committee and 
the Assessment Committee to collaborate. Because standards must be devel- 
oped to meet the technical and legal requirements of assessment and certifi- 
cation, up-front coordination is critical. For example, the research methods 
may need to be modified to meet the needs of assessment. Early coordina- 
tion will save the time, money, and effort associated with trying to correct 
problems in the standards after the fact. To ensure cooperation: 

ea A small set of individuals could serve on both committees. 

□ The committees could designate delegates to serve as liaisons between 
the two committees. 

H At a minimum, leaders or representatives of all the operating committees 
should be part of your overarching Decision Council to ensure consis- 
tency and continuity of the system as a whole. 
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Tips for Success 



1 . Ensure that key stakeholders from your Voluntary Partnership 
are represented on your assessment and certification committee. 

2 . Look for members who have: 

L^l Assessment and certification experience (e.g., human 
resources personnel, educators, representatives of unions 
that have certification programs, individuals with 
employment assessment background in your industry); 

Employment law experience (e.g., attorneys with spe- 
cialized experience in civil rights and labor law); and 



rZfl A strong background in business (e.g., members who 
may not know a lot about assessment and certification 
but who know how to develop and market products and 
services, a key part of building a successful assessment 
and certification system). 



Questions for You 

H Who do you think needs to serve on your assessment 
committee? 




i'f 



0 Which members of your Voluntary Partnership have experience 
in assessment and certification? In developing and marketing 
products and services? 

0 Do you need to make other changes in your organizational 
structure to accommodate the needs of assessment and certifi- 
cation? 

0 What do you think is the best way to coordinate activities 
between those working on developing standards and those 
working on assessment and certification? 

0 Have you included individuals who represent the interest of 
protected populations? 
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Begin Working with the NSSB on Assessment 
and Certification 



You already have a working relationship with the NSSB and its Technical 
Assistance Team. Now is a good time to discuss how you can build on that 
relationship to begin working on assessment and certification. Keep in mind 
that the NSSB is available to provide ongoing advice and assistance to help 
you carry out the steps in this manual. 




Build Relationships with Outside Groups 

Start meeting with organizations that may want to align with your certifica- 
tion program. Consider establishing “user groups, ” made up of specialty 
standards developers; employers; workers; community colleges; state educa- 
tion, training and employment agencies; unions; and civil rights and com- 
munity-based groups. Their input at key moments in the design and plan- 
ning of your program could help you build good will, produce a better 
product, and open up new markets and partnerships. 




Hire an On-Staff Assessment Expert 

The NSSB strongly recommends that you hire someone with assessment 
expertise to serve on your staff, either full time or in a consulting capacity. 
This individual (or group of individuals) can help: 



0 Coordinate work between the Standards and Assessment Committees', 



0 Oversee research to help your Voluntary Partnership learn about assess- 
ment and certification in your industry sector; 

0 Help prepare a request for proposals (RFP), evaluate bids from vendors, 
and manage the contract; and 

0 Serve as a liaison to NSSB staff, the Technical Assistance Team, and to 
the NSSB committees working on assessment and certification issues. 
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Tips for Success 

1 . When hiring assessment experts, look for such qualifications as: 

3a level of psychometric expertise that would typical- 
ly be evidenced by a Ph.D. in such fields as psychomet- 
rics, quantitative psychology, industrial psychology, 
applied statistics, educational measurement, or educa- 
tional psychology; 



Expertise in work performance measurement; 

Expertise in job/work analysis; 

^fl Experience developing and administering large-scale 
assessment programs; 



Expert knowledge of employment and civil rights 
laws and related court decisions (including Civil Rights 
Acts, American with Disabilities Act (ADA), and the Age 
Discrimination in Employment Act (ADEA)); 



Expertise in employment assessment validation prin- 
ciples; 

Experience costing the development and administra- 
tion of assessment tools; and 

9 Experience with certification or licensure programs. 

2 . If your Voluntary Partnership engages a consultant to fill this 
role, ensure that the consultant will be available to provide 
ongoing, in-depth assistance. 



O 
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Align Standards and Assessment 

As mentioned earlier, the standards must be developed to meet the technical 
and legal requirements of assessment. This process will require the help of 
someone with considerable assessment expertise. Individuals who are 
responsible for ensuring that the standards are aligned with assessment may 
find it useful to observe the data collection process to learn more about the 
skill standards; participate in the debriefing sessions after data collection; or 
regularly review standards as they are being developed. 



Tips for Success 
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1 . At each stage of the standards development process, an assess- 
ment expert should; 

Review the data collection materials (e.g., demo- 
graphic forms, population sampling plans) to ensure that 
the number and makeup of those participating in the 
research meet the technical and legal needs of the assess- 
ment system. 

a Provide feedback to the Standards Committee on 
any anticipated problems with the content of the stan- 
dards. This is especially important when the skill stan- 
dards are in draft form so that issues can be addressed 
before the standards are finalized. 
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2 . Consider gathering information through the standards develop- 
ment process that will help in designing assessments. You may 
want to ask participants in the standards development process 
such questions as: Where did you receive training? Is there a 
need for additional training in your industry? What kinds of 
continuing education opportunities are available? What kinds 
of certification do you have now? What problems did you expe- 
rience in developing your skills for this industry? What prob- 
lems did you encounter in getting certified? 
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Appraise Your Capacity 

Developing and maintaining an assessment and certification system is a 
highly technical science, a creative art, and a substantial business venture. 
Before you embark on investigating your assessment options, it is wise to 
evaluate your Voluntary Partnership’s capacity to plan and implement an 
assessment and certification system. 
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Tips for Success 

The following self-appraisal will help members of your 
Voluntary Partnership identify what they know about assess- 
ment and certification and what they still need to learn. 
(Consider building a “capacity and knowledge inventory” 
record as members of your Voluntary Partnership answer these 
questions. You can build on it as you start to investigate — the 
next step in this process.) 

How much do you know about the successful assess- 
ment and certification approaches in your industry sector? 



n a Are you familiar with the legal requirements sur- 

V| 

rounding assessment and how those requirements may 
% differ depending on the context and use of assessment? 



m 



a Are you familiar with the types of assessment ven- 
dors currently working in the marketplace? 

a Do you have ready access to technical experts in 
assessment development? Do they have experience in 
working in a variety of stakeholder contexts (e.g., com- 
panies, educational organizations, public and private 
settings)? 




a Are you familiar with the potential customers of 
your standards-based assessments and certification sys- 
tem? Do you have information about how and when 
they are likely to want access to assessment? Do you 
know what your customers will pay for assessment? 
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lZI Do you have the expertise needed to craft a commu- 
nications, marketing, and sales plan that will ensure the 
success of your assessment system? 

a Do you have the knowledge and skills needed to create 
and maintain a national database for storing results? Who 
in your voluntary partnership has created and maintained 
such databases? 



p! 

Si 

fM 



Si Do you know how to identify and address the barri- 
ers to access that might occur? 



Don’t worry if your answers reflect gaps in your knowledge. With 
the help of the NSSB, your own team of experts, and the investiga- 
tion recommended in the next section, you will quickly begin to 
build your knowledge. 
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Investigate 

Learning about assessment and certification, 
your customers, the market, civil rights and 
access concerns, and other key issues 



Now it’s time to investigate your options thoroughly, learn more about your 
customers, and think creatively about your industry’s assessment needs and 
market opportunities. In short, it’s time to do some “market research. ” 



Your findings at this stage of the process will help you begin to make concrete 
plans and build a vision of how you want your assessment and certification sys- 
tem to work. To do this, the NSSB recommends that you: 




Learn About Your Customers and Their Needs 

Learning about your customers and their needs is critical to developing the 
right mix of products and services. In fact, it may be the single most impor- 
tant aspect of your investigation. 



Start by defining exactly who your customers will be. Who will be using the 

assessments and certifications you develop? Who will be affected by them? 

These individuals are your customers and they include: 

BE! Assessment and certification candidates — students, new entrants to your 
industry sector, seasoned workers, and others, who will be assessed and 
certified (note how many different types of assessment candidates are in 
this list — each with their own needs) ; 

HI Employers — the companies and organizations that will make decisions 
based on your assessment system; 
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□ Education and training organizations — institutions that might use assess- 
ment and certification results to determine what training or education 
an individual needs; 

□ Unions — organized labor organizations that might create programs in 
support of your assessment and certification systems; and 

13 Specialty standards groups — developers of specialty standards that might 
link their assessment and certification systems to your own. For example, 
it may be that individuals who receive Core Plus Certificates would need 
to take only a modified assessment to achieve a particular specialty cer- 
tificate or vice versa. 



Once you know who your customers are, create “customer profiles” for each 

of these groups. 

For assessment and certification candidates, you might want to find out: 

D Demographic information: The size and location of your assessment 
candidate population; average age, experience, employment status, earn- 
ings, education and training, degree of linguistic diversity, and degree of 
access to the Internet. 

El The ways in which applicants, employees, and students would use assess- 
ments and certifications. Would they use assessment and certification 
results to make training and education choices? To make career choices? 
To get jobs? How do their needs differ? 

B The products and processes candidates use now to assess their perform- 
ance and communicate their qualifications to employers and others. 
What do they like and dislike about what they use now? What improve- 
ments would they like to see? 

E3 Reasonable costs candidates would be willing to pay for assessment and 
certification. 5 



5 To some extent, the ability to get a good answer to this question depends on the nature of the assessment and certification 
products and services you create. The more useful those products and services are to your potential customers, the more 
willing they may be to spend time and money to obtain them. To answer this and other similar questions, you may wish 
to describe different assessment and certification options and their benefits and then ask potential candidates how much 
time and money they would be willing to spend for each option. 
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□ Reasonable amount of time candidates would be willing to spend on 
assessment and certification. What’s a reasonable distance for them to 
travel to be assessed and certified? Where are the best places to offer cer- 
tification? 

For employers , you might want to ask: 

BS What products and processes do employers use now to evaluate potential 
new hires? Are they satisfied with these products? 

H Do they currently assess employee skills in order to target training and 
development and how much demand would there be for tools that 
would help them do that? 

i! What would they like out of a national assessment and certification sys- 
tem? 

O Are there differences in the assessment and certification needs of large 
employers versus smaller ones? If so, what are they? 

ES Under what conditions would employers pay or help to pay for employ- 
ees to be assessed and certified? 

IS What are the special needs of each subindustry within the sector? 

m What is the demand for assessment and certification for each concentra- 
tion? Is there a particularly pressing need for one concentration compared 
to another? (This will help you prioritize assessment development.) 



Tips 



for Success 




1 . Communicate with members of your Voluntary Partnership to learn 
more about the likely customers of your assessment and certifica- 
tion system. 

2 . Consider convening user groups to find out more about your cus- 
tomers and their needs. 

3 . Create customer profiles for all your customers. For example, the 
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needs and uses of the certification system are likely to be very dif- 
ferent for job applicants as compared to incumbent workers. 



Questions for You 

ES Who do you think are the main customers of your assessment and 
certification system? 

El What would you like to know about them? 

E3 What’s the best way for you to gather this information? 




Investigate Current Products and Services 

Now it’s time to investigate other assessment and certification systems. 

Work with the NSSB, other Voluntary Partnerships, and your own in-house 
assessment experts to identify and catalog best-practice assessment and certi- 
fication programs that are successful in and out of your industry sector. 

Find out how these assessments work: 



H Who designed them? 

H Who pays for them?* 

H How do they address important 
civil rights concerns? How are 
access and equity needs (e.g., lan- 
guage, cost, geography) 
addressed?* 

H What kinds of legal challenges 
have they withstood? 

a How are they marketed?* 

d How are they delivered?* 

a How are they operated and maintained? 

a How are certifications earned? What types of assessment tools do they use? 



* Investigate 

Items on this page and 
the next that have an 
asterisk next to them 
are explained in greater 
detail in the section 
entitled "Investigate" at 
the end of this chapter. 
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B What is their appeals process? 

H Are credits awarded for those who pass certain components of the assess- 
ment? If so, what are the advantages and disadvantages of this approach? 

H Are the credentials and earned certifications portable in any way? If so, 
how? 

B Why are existing selection and certification systems rejecting candidates? 
(This will give you insight into why these training, education, selection, 
and certification processes may or may not be working.) 

m How do they address practical issues, such as providing information 
about assessments, providing feedback, storing results, pricing, and 
assessment security? 

0 How quickly do they inform candidates whether they have passed or 
failed? 

To obtain this information, search the Internet, talk to the leaders of these 
programs, visit assessment centers, participate in briefings, go through sam- 
ple assessment processes. Coordinate your efforts with the NSSB, which will 
also be conducting similar benchmarking research to share with all the 
Voluntary Partnerships. 

As you investigate, ask yourself questions that will help you translate what 
you are learning into a vision of how your own assessment system will work. 
These include such questions as: 

B What mix of products and services will best meet the needs of your cus- 
tomers? 

E3 What parts of existing programs do you want to emulate? What do you 
want to avoid? 

0 What opportunities do you see for collaboration and partnerships?* 

How can you link your efforts to existing public and private education 
and training and employment systems that currently prepare people for 
work (e.g., government job centers, apprenticeship programs, communi- 
ty colleges, high schools, and vocational schools)? 

0 Given what you have learned from other programs, what are some of the 
key issues you need to address regarding the cost, accessibility, and mar- 
keting of your assessment and certification system? 
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Whenever possible, the NSSB encourages you to avoid reinventing the 
wheel. Learn from what’s working. Keep in mind that your investigation is 
just as likely to give you information about approaches you will want to 
avoid as those you will want to emulate. 




Tips for Success 

1. Start with a NSSB training session designed to give your Voluntary 
Partnership information on cutting-edge assessment options, legal 
issues, and models for assessment and certification systems. 

2 . Look more broadly than at “assessment for certification” per se. 
Investigate how employers, trainers, unions, and individuals evalu- 
ate competence and communicate it to others — even if certifica- 
tion is not a part of the process. For example, you might examine 
applicant screening processes, performance review systems, training 
provider entrance and exit assessments, and other aspects. These 
programs may offer interesting insights. 

3 . Look for market leaders — where all stakeholders are being served — 
and learn from them. 



Questions for You 

0 What’s the best way to find out more about the assessment and cer- 
tification market within your industry sector? 

13 What else would you like to find out about the way other products 
and services are designed and delivered? 
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Investigate— Marketing 

You will be introducing a new product to the market. Most employers and 
potential assessment candidates have worked for years without the benefit 
of assessment and certification. You can expect many questions from 
customers: 



ES Workers may say . . .“I have always just thrown together a resume or 
filled out an application to get a job. Why should I bother going 
through the time and expense of assessment and certification?” 

11 Employers may say . . .“I trust my instinct when I hire folks. Why do I 
need your help?” 

B Trainers may say . . .“I know what people need. Am I really going to 
find out more from your assessment system?” 

The importance of marketing this new system — educating people about 
how it will work and highlighting its benefits — is critical. What do you 
think is the best way to reach different customers? How are other organiza- 
tions marketing their programs? How do they help people access informa- 
tion about their program? Do they reach customers through other pro- 
grams? For example, do they work with community colleges? How can you 
draw on lessons learned from your Voluntary Partnership’s marketing com- 
mittee and work with them and your assessment vendor to answer these 
questions? Investigate. 




Investigate — Delivery Channels 

What channels will you use to deliver your assessment and certification 
products and services? There are many options, including: 



B Independent assessment centers run by your Voluntary Partnership; 

B Existing public job centers; and 

B Licensing of the assessment and certification process to third parties. In 
such a situation, your Voluntary Partnership would develop the assess- 
ment tools, but others (community colleges, employers, unions, special- 
ty standards providers, and others) would administer and possibly score 
them. Your Voluntary Partnership would still award the certificates and 



GETTING STARTED ON ASSESSMENT: Developing a Voluntary System of Assessment and Certification Based on Skill Standards 31 



would oversee licensees to ensure quality. This may be more suitable as a 
long-term option, once your Voluntary Partnership gains more direct 
experience and fine-tunes processes. Also, it will be important to consid- 
er any potential conflicts of interest that may arise as a result of such 
arrangements. 

How do other certification programs address this issue? What do they see as 
the benefits and drawbacks? Investigate. 




Investigate — Financing 

Developing and maintaining an assessment and certification system, partic- 
ularly one as large as the one you will be developing, can be very costly. 
How do other certification systems pay for themselves? How do they save 
money? Do they tend to get back their costs through fees charged to assess- 
ment candidates? Or do they find other revenue sources? When you speak 
to employers, be sure to find out how much they would be willing to con- 
tribute to the development and ongoing maintenance of the system. What 
would they want in return? Can you think of any innovative ideas for how 
to finance the system? Investigate. 




Investigate — Operations and Maintenance 

A lot goes into operating and maintaining an assessment and certification 
system. Some important issues to consider include: 



□ Database management — Records of assessment scores and certificates 
will need to be stored electronically in a national database. This database 
will need to link to the NSSB national database. 

H Recruitment and training of assessment administrators. 

E3 Continuous updating of assessments for security reasons. 

O Establishment of policies ranging from how to respond to complaints 
and legal challenges to how to handle ambiguities in scoring. 
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A lot of these issues will need to be addressed by your assessment team in Step 
Four, but keep track of what’s working and what’s not in order to share with 
your assessment team and to help you evaluate your options. Investigate. 




Investigate — Partnerships 

Developing strategic partnerships with specialty skill standards developers, 
educators, and trainers is an important part of building a successful assess- 
ment and certification system. Which committees and individuals within 
your Voluntary Partnership are most suited to do this work? How will your 
assessment team work with them to get this done? How can the NSSB and 
other Voluntary Partnerships help you leverage resources? Investigate. 




Investigate — Civil Rights Issues 

Creating a system that ensures equality, fairness, and access for all 
Americans needs to be one of your top priorities. When looking at other 
programs, find out how they have successfully addressed these issues. 



It will be important for your Voluntary Partnership to communicate to 
employers that they are legally required to conduct an internal validation of 
the standards and related assessments before using them to make hiring and 
promotion decisions. Many employers will already be familiar with this 
process, as they often carry it out when they use other new standards or 
assessments. The purpose of the internal validation is to ensure that the 
knowledge, skills, and performance described by the standards are required 
for the job. 



The NSSB will be developing tools to help employers in your industry sec- 
tor quickly and cost-effectively validate the standards. Please note that even 
those employers who cannot validate that all the skill standards are neces- 
sary for work in their organization may still be able to use part of the Core 
Plus Certificates, particularly if your Voluntary Partnership has developed 
assessments in modules. With a modular system, your Voluntary 
Partnership could award assessment candidates credit for achieving different 
modules, which employers can then use as a factor in making hiring deci- 
sions once the necessary validation has been conducted. 
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Envision 

Creating a vision of how you want your 
assessment and certification system to work 



You have a strong sense of your customers, their needs, and the market. Now it’s 
time to start to envision how you want your assessment and certification system 
to work. This is your chance to apply what you have learned and think creatively 
about how to solve key challenges and address important customer needs. 

To do this, the NSSB recommends that you: 




Set Goals 

Consider starting this effort by setting some broad goals for your assessment 
and certification system — perhaps creating something like a corporate mis- 
sion statement. What are you trying to achieve? Five years from now, how 
will you know you have succeeded? 



Your goals should reflect the unique needs of your industry sector. They 
should also reflect the objectives expressed in the National Skill Standards 
Act. In that legislation, the NSSB and the Voluntary Partnerships were 
charged with developing a voluntary national system of standards, assess- 
ment, and certification that will: 

B Provide individuals with portable certificates they can use across an 
industry sector; 

S Raise the level of workforce skills; 

13 Increase job security for frontline workers; 

E Streamline training, education, recruitment, and hiring; 
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□ Increase opportunities for minorities and women; 

□ Boost the competitiveness and effectiveness of U.S. employers; 

□ Increase retention rates and reduce turnover; and 

□ Prepare individuals for the demands of the high-performance workplace. 

As you build your vision, you should be able to identify how your plans will 
help achieve these goals. Remember the process of clearly defining your 
goals and objectives at the outset will help guide the decisions you make 
over time. 




Develop a Vision 

You don’t need every detail worked out at this stage. But you do need to 
start building a vision that will guide the development of your system. Use 
the following questions to get you started. And pose a few of your own! 



Delivery 

H How will you deliver your products and services (e.g., at assessment cen- 
ters, at community colleges, through a licensing agreement)? 

B In what key cities or towns do you plan to make the assessment and cer- 
tification available? 

E What degree of access to the system do you want to achieve? For exam- 
ple, available in every city and state? Access over the Internet? Through 
schools? At company sites? At assessment centers? At night? On the 
weekends? 

H How will different assessment candidates access information about your 
system? 

B How can you ensure access to your system (e.g., geographic access, access 
for the disabled, access for those of limited means, access for those who 
do not speak English)? 
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Marketing 

E3 How will you work with your 
Voluntary Partnership’s market- 
ing committee to promote the 
system to . . . 

Students? 

New entrants? 

Incumbent workers? 

Large employers? 

Smaller employers? 

Training organizations? 

Human Resources personnel? 

Specialty standards groups? 

States? 

IS What kinds of education and 
communications programs will 
you need? For example, many 
employers in your industry sector 
will know very little about your 
assessment and certification sys- 
tem, particularly at the beginning. 

You may need to develop an edu- 
cational strategy to help human 
resources personnel and managers 
interpret certificates. 

Partnerships 

B8 What partnerships do you need 
to create in order to ensure suc- 
cess? 

ES How can you link your system to education and training? 

B How will you link your system with specialty skill standards developers? 



Think Big 

You probably noticed 
that these questions do 
not relate to the devel- 
opment of assessment 
tools (e.g., Should indi- 
viduals be assessed 
using a multiple-choice 
test or a work simula- 
tion? What kind of 
diagnostic feedback 
should be provided? 
Should you use video 
technology?). 

These are important 
questions, but first it's 
important that you con- 
sider some big picture 
issues. Many of these 
larger issues — from your 
budget constraints to 
the ways in which you 
deliver the system — will 
influence the types of 
assessment tools you 
develop. In addition, 
development of the 
assessment tools will 
require the assistance of 
your vendor, the focus 
of Step 4. 
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Finance 



S3 How will you finance the development and ongoing maintenance of 
your assessment and certification system? 

d What is the maximum per-candidate fee you will charge? 

13 What is the maximum you can spend on developing and maintaining 
the system? This depends greatly on the financing available to your 
industry sector. As you consider this issue, it’s important to identify the 
available support within your sector for various aspects of research as 
well as how much employers and educators are willing to pay for the 
development and operation of the system. 



Tips for Success 

1 . Make sure your vision meets the needs of your market and the 
NSSB criteria. For example, does your vision: 

7A Reflect the needs of your customers (e.g., large employers 

versus smaller ones, subindustry groups, students, new entrants, 
and seasoned workers)? 

7A Address issues of equity and access? 

7A Comply with civil rights and employment law? 

71 Ensure portability of certification across your industry sector? 

7A Reflect the need to be practical and to create a system that is 

affordable? 

2 . Develop hypothetical case examples, similar to those offered in the 
last chapter of Credentials for Success, to help you develop and 
communicate your vision. These examples, particularly if you base 
them on the circumstances and the needs of real customers, will 
help you think through some the details that are so critical to the 
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success of your system. The case examples may also help you com- 
municate your plans to others, including employers, assessment 
candidates, and the press. 

3 . Be realistic. Know your constraints (e.g., a limited budget, a maxi- 
mum per-candidate-fee) . As you develop your vision, you need to 
make the necessary tradeoffs in order to create a practical, efficient, 
and cost-effective system. For example, your Voluntary Partnership 
may want to deliver certain services over the Internet, but the up- 
front costs may be too high for you to do so immediately. 

Questions for You 

B Who do you need to involve in this process? From whom do you 
need to get buy-in? 

ES What will you need to do to start turning these ideas into reality? 
Who will be in charge of carrying out each step? 

S Does your vision meet the goals you outlined at the start of this 
process? 
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Plan and Retain 

Retaining expert assistance to plan, develop, 
administer, and maintain the system 



Once you have a clear idea of what you want your system to achieve, you are 
ready to begin working with a vendor or vendors to develop the system (e.g., 
the actual assessments, delivery channels, a database for storing results). 

Developing assessments that truly measure what someone knows and can do is 
difficult. To ensure the quality of your assessments and their technical and 
legal validity, we recommend that you work with leading experts during this 
phase of the work. 

This last point — the need to meet technical and legal standards — cannot be 
stressed enough. The assessments you develop will be used to make high-stakes 
decisions about hiring and placement. That’s a huge responsibility, with major 
implications for the careers of individuals and the competitiveness of U.S. 
companies. It also means that your Voluntary Partnership could face legal 
challenges. 

The best way to ensure the quality of your assessment system is to work with 
assessment experts and people who have experience in employment and civil 
rights law — a key requirement of the NSSB. In this chapter, we provide you 
with information designed to help you retain and begin working with vendors 
to develop your system. 

Keep in mind that you will not be alone. The NSSB plans to work in partner- 
ship with you and your vendor to help develop the system. The NSSB 
Endorsement Review Panel (ERP) will also be working with you and your ven- 
dor, reviewing key plans and products at critical stages of your work in order to 
ensure they comply with NSSB criteria. 
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This chapter focuses on how to retain the help of a vendor. To retain a vendor, 
the NSSB recommends that you: 




Gain an Understanding of the Assessment 
Development Process 

Your Voluntary Partnership does not need to become expert in every aspect 
of assessment, but you do need to understand the key activities involved in 
the process. Without an understanding of these activities, it will be virtually 
impossible to develop a meaningful request for proposals (RFP), select a 
qualified vendor, manage that vendor, or make valuable contributions along 
the way. 



The following section describes the key steps involved in the development 
of an assessment and certification system once you have hired your vendor. 
Your goal at this early stage is not to carry out these activities — that’s the job 
of the vendor you hire. Instead, spend some time gaining a general under- 
standing of what’s involved so that you can prepare your RFP. The NSSB 
will be providing more in-depth guidance for many of these issues when 
you and your vendor begin developing the actual system. 



Please note: The following section — An Overview of the Process — provides 
information you can use in your RFP to explain what will be required of 
vendors. It is similar to the information provided in the Introduction of this 
publication but goes into greater depth. 



An Overview of the Process 

m Activity One: Gain an in-depth understanding of the skill 
standards as well as the project goals, constraints, and 
parameters. Your vendor will need to learn about: 

1 . The goals of the national system. What is the NSSB? What are 
the goals of the national system? How is the system organized 
(e.g., core, concentration, specialty, information on industry 
sectors)? How does the NSSB endorsement process work? 
What are the NSSB criteria? We recommend you set up brief- 
ing sessions among NSSB staff, your Voluntary Partnership, 
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and your vendor so that your vendor can ask questions 
directly and learn firsthand about the project. 

2. Your vision of assessment and certification. What are your 
Voluntary Partnership's goals, parameters, and constraints 
(i.e., the vision you identified in Step Three of this publica- 
tion)? Your Voluntary Partnership needs to provide the 
vendor with information about any decisions that will 
affect the development of the assessment tools, such as: 

a Product plans; 

a Budget constraints; 

a The size, geographic dispersion, and demographic 
diversity of the candidate population; 

a Available support within the sector for various aspects 
of research and ongoing assessment (e.g., One of the 
key steps later in the process is the pilot testing of 
assessment tools. The vendor will need to know if you 
can help them access workers and other assessment 
candidates in order to conduct these pilots.); 

a Objectives with regard to access to the system and lin- 
guistic diversity; 

a The maximum fee permissible for assessment candi- 
dates; 

a A profile of your customers, including detailed infor- 
mation on assessment candidates, and anything you 
learned about the market (i.e., the results of the inves- 
tigations you carry out as described in Step Two of this 
publication); and 

a Who will be developing the implementation plan and 
carrying out assessment administration? 

3. Your Voluntary Partnership's skill standards. The following 
review materials should be available to your vendor: 

a The academic, employability, and occupational and 
technical knowledge and skills and complexity levels; 
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raThe list of critical work functions, key activities, and 
performance indicators and their associated academic, 
employability, and occupational and technical knowl- 
edge and skills; 

EaAny underlying data, including results of the national 
standards survey; and 

eh Information on the concentrations and the core skill 
standards. 

Questions your vendor will want to consider include: How 
much variability exists across the concentrations and what 
are the implications for assessment? (i.e., Do the knowl- 
edge and skill levels required differ widely across concen- 
trations? Do they differ widely across critical work func- 
tions within a concentration? Can one assessment be used 
to cover more than one concentration?) 

o Activity One: This activity will be covered 
later in the book. 



0 Activity Two: Develop a Preliminary 

Assessment System Plan. This is the first step 
in developing assessments and involves the 
important and challenging task of determin- 
ing which aspects of the skill standards will 
be assessed and the best methods for assessing them. Your 
Preliminary Assessment System Plan should provide informa- 
tion on the estimated importance assigned to each aspect of 
the standards, the proportion of the assessments that will be 
dedicated to each component of the skill standards, and an 
estimate of scoring and examination time. It should also pro- 
vide a vision of how the various aspects of the plan will lead 
to certification, how the system will be delivered, what system 
components are needed, etc. Your Preliminary Assessment 
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System Plan, Deliverable #2, must be submitted to the ERP for 
review prior to ultimate NSSB approval. 

This plan is the first of several plans your vendor will need to 
develop on the way to building the assessment system. Two 
of the most challenging technical tasks involved in develop- 
ing the plan will be determining: 

m The content that needs to be measured; and 

m The methods that will be used to measure this content. 

Let's look at each of these tasks for a moment, keeping in 
mind that the NSSB will be providing more detailed guidance 
to each Voluntary Partnership and its vendor. 

1 . Determine the Content to Be Measured. The sheer volume 
of information contained in the standards could make the 
time and money it would take to measure every aspect 
prohibitive and impractical. Although it may be possible to 
develop assessment tools that measure all of the require- 
ments of core and concentration skill standards, your ven- 
dor will likely need to prioritize which aspects of the core 
and concentration skill standards to assess as part of the 
assessment process. 

When making these choices, your vendor will need to con- 
sider budget, relative importance of particular aspects of 
the standards, and measurement feasibility. The most 
important content issue your vendor will need to consider 
is how to ensure the content validity of the assessment 
and certification process. By content validity, we mean the 
degree to which the assessment and certification system 
you develop reflects the content of the skill standards. 
Determining the content to be measured will include such 
steps 7 as: 

m Analyzing the skill standards for core and each concen- 
tration to determine where overlap exists. 



7 These major steps are described in general terms, and it should be noted that ensuring content validi- 
ty is a difficult process that requires a high degree of assessment expertise and experience. 
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■ Prioritizing the importance of the standards for core 
and concentration. This can be done in a variety of 
ways. For example, if your Voluntary Partnership con- 
ducted a national survey in which respondents rated 
the importance of different aspects of the standards to 
their jobs, those findings could be one of several 
sources of information your vendor could use to make 
these determinations. 

m Determining what to measure based on threats to 
validity, importance of a particular aspect of the stan- 
dards, measurement feasibility, cost, and others. 

Keep in mind that the assessments must capture the full 
richness of the skill standards. After all, skill standards are 
made up of a considerable amount of information, includ- 
ing the work-oriented component (e.g., the critical work 
functions, key activities, and performance indicators) and 
the worker-oriented component (e.g., the academic, 
employability, and occupational and technical knowledge 
and skills needed to carry out the critical work functions). 

If assessments measure only knowledge and skills divorced 
from the context of work described by the critical work 
functions, they will have limited value. The assessments 
may tell us whether individuals are good at mathematics 
or have strong listening skills, but they will not necessarily 
tell us whether those individuals can use those skills in the 
context of work. 

That's why assessments developed based on skill standards 
should measure the knowledge and skills in the context of 
the critical work functions, key activities, and performance 
indicators whenever possible. For example, instead of meas- 
uring an individual's writing skills or speaking skills, the 
assessments should measure whether someone can use 
those skills to identify a customer need (e.g., identifying 
customer needs is the focus of one of the manufacturing 
critical work functions). 




46 GETTING STARTED ON ASSESSMENT: Developing a Voluntary System of Assessment and Certification Based on Skill Standards 

43 



Another major challenge in identifying the content to be 
measured will be to do so in a way that factors in modular- 
ity. By modularity, we mean the development of assess- 
ments around clusters of related skill standards. For exam- 
ple, to obtain a Core Plus Certificate, individuals may need 
to be assessed in standards covered by eight different criti- 
cal work functions. In a modular system, one assessment 
module might focus on one critical work function, along 
with its associated key activities, performance indicators, 
and knowledge and skills, while another module might 
focus on another critical work function. 

Modularity gives the Voluntary Partnerships flexibility in 
the kinds of products and services they can create. For 
example, individuals can work toward certification over 
time, completing modules along the way to earning a 
Core Plus Certificate. 

Modularity also means that instead of requiring individu- 
als to retake the entire assessment, Voluntary Partnerships 
will be able to allow candidates to apply passing results in 
certain modules to future efforts. This will require fairly 
detailed record keeping but could serve as an incentive for 
some individuals to continue working toward certification, 
saving candidates time and money. Some Voluntary 
Partnerships may wish to take this one step further by 
offering a record of achievement so that individuals can 
demonstrate to employers their success in different aspects 
of the assessment. 

Modularity will also help the Voluntary Partnerships pro- 
vide more detailed feedback (e.g., feedback on how an 
individual performed on a given module). 

Equally important, modularity will help promote portabil- 
ity across different industry sectors (i.e., When assess- 
ments are modular, it's easier to see the similarities across 
industry sectors. If there are similarities, it may be possible 
for individuals to use certification or a part of their certifi- 
cation earned in one industry sector to demonstrate qual- 
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ifications in another industry sector). In addition, the 
greater degree of modularity in assessments, the easier it 
will be for candidates to transfer credits earned for one 
concentration to another (i.e., It's possible that there will 
be overlap between the standards required for one con- 
centration and those required for another. In that case, 
the more modularity there is in the system, the easier it 
will be for individuals to apply credits from one assess- 
ment to another). 

2. Identify the Assessment Methods to Be Used. Assessment 
methods can range from paper-and-pencil multiple- 
choice assessments to work simulations to observations 
of an individual's on-the-job performance. There is no 
one best method. 

NSSB criteria encourage the use of innovative methods of 
assessment and assessment delivery as long as the meth- 
ods are consistent with all other NSSB criteria. 

Innovative assessment methods include live performance 
simulations. These types of assessments, which measure an 
individual's knowledge and skills based on how they per- 
form a simulated work task, are often useful for measuring 
social or interpersonal skills, such as listening, speaking, 
working in teams, building consensus, and using social skills 
(which fall under the category of employability skills within 
the NSSB framework). 

For example, an assessment might involve a simulation of 
a customer calling about a complaint. The examinee com- 
pletes a form using information gained from the caller. To 
complete the form, the examinee might access a computer 
file to obtain certain codes, use some basic arithmetic cal- 
culations, and write a short narrative description of the 
problem. Following completion of the form, an examiner 
might ask the examinee a series of structured questions 
designed to assess his or her understanding of the prob- 
lem. Key considerations in this type of simulation are the 
need to provide a standardized stimulus (i.e., the customer 
problem) and a standardized scoring system. 
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Other performance simulations might involve such assign- 
ments as assembling a hypothetical machine or product 
based on a set of written instructions; writing a report 
based on a set of documents; or taking action on a series 
of problems appearing in a hypothetical in-basket (these 
could be delivered via e-mail). The creative use of video 
and computer technology provides for the development, 
delivery, and scoring of a much wider array of simulations 
than was previously available. 

Voluntary Partnerships may also wish to use a "situational 
judgment inventory." This is a method of assessment in 
which examinees are asked to consider various hypothetical 
work situations. The examinee is provided with a series of 
alternative actions that he or she might recommend. This 
method may be more practical, but it is a less realistic way 
of assessing many interpersonal and problem-solving skills. 

Another type of innovative assessment, albeit more expen- 
sive and more cumbersome to administer to large groups of 
people, might involve live role-plays designed to assess 
interpersonal or social skills. Such an assessment might 
include a group exercise in which an individual's skills at 
building consensus are evaluated, or a situation in which an 
individual's ability to close a real estate sale is evaluated. 

This stage of assessment might also include a videotape of 
the performance of some sample of the examinee's work, 
which is then rated by a group of trained experts using 
standardized rating criteria and instruments. 

The choice of assessment methods will depend a great deal 
on the specific aspects of the standards being measured, how 
well those assessments measure performance, as well as prac- 
tical considerations of time and expense. The most important 
goal in choosing assessment methods should not be whether 
they are innovative or not, but whether the method selected 
is the best method for measuring the content. 

The plan should include not only a list of the content to be 
measured and the assessment methods used, but also infor- 
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